Image analysis apparatus and monitoring system

ABSTRACT

An image analysis apparatus includes a person information acquisition unit configured to analyze a captured image of a monitored space imaged by a camera so as to acquire information of a person imaged in the captured image, and a hidden state detection unit configured to detect occurrence of a hidden state in which a plurality of the persons are imaged to overlap in the captured image.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is based on and claims priority under 35 U.S.C. § 119to Japanese Patent Application 2021-098861, filed on Jun. 14, 2021, theentire content of which is incorporated herein by reference.

TECHNICAL FIELD

This disclosure relates to an image analysis apparatus and a monitoringsystem.

BACKGROUND DISCUSSION

In the related art, an image analysis apparatus is known that analyzes acaptured image of a monitored space imaged by a camera so as to acquireinformation of a person imaged in the captured image. For example, JP2020-104680A (Reference 1) discloses a configuration in which a vehicleinterior of a vehicle is set as a monitored space, and a posture and aphysique of an occupant imaged in a captured image by a camera aredetected.

However, depending on a positional relationship with the camera, aperson located in the monitored space may not be correctly imaged in thecaptured image. Thus, acquisition of information of the person may behindered.

SUMMARY

According to an aspect of this disclosure, an image analysis apparatusincludes: a person information acquisition unit configured to analyze acaptured image of a monitored space imaged by a camera so as to acquireinformation of a person imaged in the captured image; and a hidden statedetection unit configured to detect occurrence of a hidden state inwhich a plurality of the persons are imaged to overlap in the capturedimage.

According to another aspect of this disclosure, a monitoring systemincludes the image analysis apparatus according to any one of the aboveaspects.

BRIEF DESCRIPTION OF THE DRAWINGS

The foregoing and additional features and characteristics of thisdisclosure will become more apparent from the following detaileddescription considered with the reference to the accompanying drawings,wherein:

FIG. 1 is a perspective view of a vehicle to which a monitoring systemis applied;

FIG. 2 is a diagram illustrating an occupant in a vehicle interior and acamera that captures an image of the occupant;

FIG. 3 is a diagram illustrating a case where the vehicle interior isviewed from above;

FIG. 4 is a block diagram showing a schematic configuration of an imageanalysis apparatus;

FIG. 5 is a diagram illustrating skeleton points of a person;

FIG. 6 is a schematic configuration diagram of the monitoring system;

FIG. 7 is a block diagram showing a schematic configuration of a hiddenstate detection unit;

FIG. 8 is a diagram illustrating movement prediction for the occupantand overlap ratio calculation;

FIG. 9 is a flowchart showing a processing procedure of hidden statedetection based on overlap ratio determination;

FIG. 10 is a diagram illustrating boarding detection;

FIG. 11 is a diagram illustrating alighting detection;

FIG. 12 is a flowchart showing a processing procedure of measurement ofthe number of boarding and alighting occupants;

FIG. 13 is a flowchart showing a processing procedure of hidden statedetection based on number-of-persons difference determination;

FIG. 14 is a diagram illustrating the occupant imaged in a capturedimage at a hiding position in proximity to the camera.

FIG. 15 is a flowchart showing a processing procedure of hidden statedetection based on camera proximity position determination;

FIG. 16 is a flowchart showing a processing procedure of hidden statedetection and abnormality detection; and

FIG. 17 is a flowchart showing a processing procedure of determinationoutput for a detection state of the skeleton points.

DETAILED DESCRIPTION

Hereinafter, an embodiment of an image analysis apparatus and amonitoring system will be described with reference to drawings.

As shown in FIGS. 1 to 3 , a vehicle 1 of the present embodimentincludes a vehicle body 2 which extends in a front-rear direction of thevehicle and has a substantially rectangular box shape. A door openingportion 3, which serves as an entrance and exit for an occupant, isprovided in a side surface of the vehicle body 2. The door openingportion 3 is provided with a pair of slide doors 4 and 4 that are openedand closed in the front-rear direction of the vehicle and in oppositedirections. An occupant 5 in the vehicle 1 is on board the vehicle 1 ina “seating posture” in which the occupant 5 is seated on a seat 7provided in a vehicle interior 6, or a “standing posture” in which ahanging strap or a handrail (not illustrated) is, for example, used.

Further, a camera 8 that captures an image of an inside of the vehicleinterior 6 is provided in the vehicle 1 of the present embodiment. Inthe vehicle 1 of the present embodiment, the camera 8 is provided in thevicinity of a ceiling portion 9 near a corner portion 6 fa at a frontposition of the vehicle interior 6. As the camera 8, for example, aninfrared camera or the like is used. Thus, the camera 8 of the presentembodiment is configured to capture an image of the occupant 5 in thevehicle 1 from a predetermined direction set in the vehicle interior 6.

As shown in FIG. 4 , in the vehicle 1 of the present embodiment, acaptured image Vd of the inside of the vehicle interior 6 imaged by thecamera 8 is input to an image analysis apparatus 10. Further, the imageanalysis apparatus 10 has a function of monitoring a state of the insideof the vehicle interior 6 imaged in the captured image Vd by analyzingthe captured image Vd. Thus, in the vehicle 1 of the present embodiment,a monitoring system 15 is constructed in which the vehicle interior 6imaged by the camera 8 is set as a monitored space 11.

In detail, the image analysis apparatus 10 of the present embodimentincludes an image analysis unit 20, and a person recognition unit 21that recognizes a person H in the vehicle interior 6 imaged in thecaptured image Vd, that is, the occupant 5 in the vehicle 1, based on aresult of image analysis performed by the image analysis unit 20. In theimage analysis apparatus 10 of the present embodiment, the personrecognition unit 21 performs recognition processing for the person H byusing an inference model generated by machine learning. Further, theimage analysis apparatus 10 of the present embodiment includes anabnormality detection unit 22 that detects an abnormality occurring inthe vehicle interior 6 imaged by the camera 8 by monitoring the occupant5 in the vehicle 1 thus recognized.

Specifically, as shown in FIGS. 4 and 5 , the image analysis apparatus10 of the present embodiment includes a skeleton point detection unit 23that detects skeleton points SP of the person H included in the capturedimage Vd. That is, the skeleton points SP are unique points thatcharacterize a body of the person H, such as joints and points on a bodysurface, and correspond to, for example, a head, a neck, shoulders,armpits, elbows, wrists, fingers, a waist, a hip joint, buttocks, knees,ankles, and the like. Further, in the image analysis apparatus 10 of thepresent embodiment, the skeleton point detection unit 23 also performsdetection processing of the skeleton points SP by using an inferencemodel generated by machine learning.

As shown in FIG. 4 , the image analysis apparatus 10 of the presentembodiment includes a feature calculation unit 24 that calculates afeature Vsp based on detection of the skeleton points SP. Specifically,in the image analysis apparatus 10 of the present embodiment, thefeature calculation unit 24 calculates, based on positions of theskeleton points SP on two-dimensional coordinates in the captured imageVd, the feature Vsp of the person H imaged in the captured image Vd.Further, the feature calculation unit 24 calculates the feature Vsp ofthe person H imaged in the captured image Vd, for example, based on abody size indicated by the plurality of skeleton points SP, such as ashoulder width of the occupant 5. The image analysis apparatus 10 of thepresent embodiment includes a person information acquisition unit 25that acquires, based on the feature Vsp of the person H obtained by aseries of analysis processing, information Ih of the person H recognizedin the monitored space 11 imaged by the camera 8.

In detail, the person information acquisition unit 25 of the presentembodiment includes a posture determination unit 26 that determines aposture of the person H imaged in the captured image Vd. The posturedetermination unit 26 of the present embodiment inputs the feature Vspof the person H acquired from the feature calculation unit 24 to aninference model generated by machine learning. Then, the posturedetermination unit 26 determines the posture of the person H imaged inthe captured image Vd of the inside of the vehicle interior 6 based on aposture determination probability value thus obtained.

Specifically, the posture determination unit 26 of the presentembodiment includes a standing determination probability valuecalculation unit 26 a that calculates a probability that a posture ofthe occupant 5 as a posture determination target person is the “standingposture”. In addition, the posture determination unit 26 includes aseating determination probability value calculation unit 26 b thatcalculates a probability that the posture of the occupant 5 as thetarget person is the “seating posture”. Further, the posturedetermination unit 26 of the present embodiment includes a fallingdetermination probability value calculation unit 26 c that calculates aprobability that the posture of the occupant 5 as the target person is a“falling posture”.

That is, in the posture determination unit 26 of the present embodiment,as the posture determination probability value, the standingdetermination probability value calculation unit 26 a calculates astanding determination probability value XA, the seating determinationprobability value calculation unit 26 b calculates a seatingdetermination probability value XB, and the falling determinationprobability value calculation unit 26 c calculates a fallingdetermination probability value XC. Further, the posture determinationunit 26 of the present embodiment performs calculation of the posturedetermination probability value such that a total value of the standingdetermination probability value XA, the seating determinationprobability value XB, and the falling determination probability value XCis “1.0”. Thus, the posture determination unit 26 of the presentembodiment can determine the posture of the occupant 5 without anycontradiction based on the posture determination probability value.

The standing determination probability value XA calculated by thestanding determination probability value calculation unit 26 a of thepresent embodiment is further divided into a probability that theoccupant 5 in the “standing posture” is in a “moving state”, aprobability that the occupant 5 is in a “still state”, and a probabilitythat the occupant 5 is in a “state in which the hanging strap, thehandrail, or the like is used”. Thus, the posture determination unit 26of the present embodiment can subdivide and determine the “standingposture”.

In the image analysis apparatus 10 of the present embodiment, when theposture determination unit 26 determines that the occupant 5 in thevehicle 1 falls down, the abnormality detection unit 22 determines thatthe abnormality occurs in the vehicle interior 6 imaged in the capturedimage Vd by the camera 8. Thus, the monitoring system 15 of the presentembodiment can ensure safety in the vehicle interior 6.

In the image analysis apparatus 10 of the present embodiment, the personinformation acquisition unit 25 is provided with an attributedetermination unit 27 that determines an attribute of the person Himaged in the captured image Vd, a physique determination unit 28 thatdetermines a physique of the person H, and the like, in addition to theposture determination unit 26. Thus, the image analysis apparatus 10 ofthe present embodiment can detect, with high accuracy, a state of theperson H imaged in the captured image Vd.

As shown in FIG. 6 , the monitoring system 15 of the present embodimentis formed by connecting a plurality of information processingapparatuses 30, which are disposed inside and outside the vehicle 1,with one another via an information communication network (notillustrated). Specifically, the image analysis apparatus 10 of thepresent embodiment is configured such that an in-vehicle informationprocessing apparatus 30 a mounted in the vehicle 1, and an informationprocessing apparatus 30 b which is outside the vehicle and constitutes acloud server 31 perform image analysis processing in a distributedmanner. Thus, the monitoring system 15 of the present embodiment isconfigured to ensure excellent mountability in the vehicle 1 by reducinga calculation load of the information processing apparatus 30 a mountedin the vehicle 1.

Further, the monitoring system 15 of the present embodiment isconfigured such that when the abnormality occurs, an administrator 35outside the vehicle, such as an operator 33 stationed in an operationcenter 32 of the vehicle 1, can confirm the captured image Vd of theinside of the vehicle interior 6 imaged by the camera 8. Thus, in themonitoring system 15 of the present embodiment, high reliability andsafety are ensured.

(Hidden State Detection)

Next, a hidden state detection function implemented in the imageanalysis apparatus 10 of the present embodiment will be described.

As shown in FIG. 4 , the image analysis apparatus 10 of the presentembodiment includes a hidden state detection unit 40 that detectsoccurrence of a so-called hidden state in which acquisition of theinformation Ih based on the image analysis may be hindered since aplurality of persons H are imaged to overlap in the captured image Vd.That is, when a plurality of occupants 5 are on board the vehicle 1, theoccupant 5 at a position away from the camera 8 may be hidden by theoccupant 5 at a position closer to the camera 8 in the captured image Vdof the vehicle interior 6 imaged by the camera 8. As a result, therearises a problem that for the back occupant 5 hidden by the frontoccupant 5, a presence thereof cannot be recognized or the detection ofthe skeleton points SP is incomplete, so that occupant information Ichcannot be acquired with high accuracy.

In view of the above description, in the image analysis apparatus 10 ofthe present embodiment, the occurrence of such a hidden state isdetected by the hidden state detection unit 40. Thus, the monitoringsystem 15 of the present embodiment is configured such that a state inwhich the occupant 5 in the vehicle interior 6 is not correctly imagedin the captured image Vd can be grasped.

Specifically, as shown in FIG. 7 , the hidden state detection unit 40 ofthe present embodiment includes a first hidden state determination unit41, a second hidden state determination unit 42, and a third hiddenstate determination unit 43. The first hidden state determination unit41, the second hidden state determination unit 42, and the third hiddenstate determination unit 43 are configured to perform, by differentmethods, determination of the hidden state occurring in the capturedimage Vd by the camera 8, respectively.

(Overlap Ratio Determination)

First, detection and determination of the hidden state based on overlapratio determination performed by the first hidden state determinationunit 41 will be described.

As shown in FIG. 8 , in the hidden state detection unit 40 of thepresent embodiment, the first hidden state determination unit 41performs movement prediction for the person H imaged in the capturedimage Vd by the camera 8. Further, the first hidden state determinationunit 41 determines, based on the movement prediction, whether there isan imaged area 50 of the person H estimated to be “imaged to overlap”after a predetermined future time. As the “predetermined future time”,for example, a scheduled timing or the like at which next image analysisis performed is set. Further, the first hidden state determination unit41 calculates an overlap ratio α of the imaged area 50 for such theimaged area 50 of the person H which is imaged to overlap. The firsthidden state determination unit 41 of the present embodiment performsthe detection and determination of the hidden state occurring in thecaptured image Vd based on a comparison between the overlap ratio α anda predetermined overlap determination value αth.

In detail, in the image analysis apparatus 10 of the present embodiment,an imaged area 60 of an upper body of the occupant 5 imaged in thecaptured image Vd, specifically, a torso portion thereof is used as theimaged area 50 of the person H in the calculation of the overlap ratio αperformed by the first hidden state determination unit 41. Specifically,the first hidden state determination unit 41 of the present embodimentcalculates a movement direction and a movement amount of a center point60 x for imaged areas 61 and 62 of the torso portion detected in thecaptured image Vd at previous and current analysis timings. Further, thefirst hidden state determination unit 41 compares sizes of these imagedareas 61 and 62. Thus, the first hidden state determination unit 41 ofthe present embodiment is configured to predict, at the predeterminedfuture time, a position and a size of an imaged area 63 imaged in thecaptured image Vd.

For example, FIG. 8 illustrates a situation in which an occupant 5 b inthe standing posture walks across a position on a front side closer tothe camera 8 than an occupant 5 a in a seated posture who is seated onthe seat 7 in a rear seat area A1. In this case, for the occupant 5 a inthe seated posture, an imaged area 50 a in the captured image Vd doesnot substantially move. On the other hand, for the occupant 5 b in thestanding posture, it is predicted that an imaged area 50 b moves fromthe lower right to the upper left in the captured image Vd in FIG. 8 asthe occupant 5 b walks. Further, an overlap area 55 in which the imagedareas 50 a and 50 b of the occupants 5 a and 5 b are imaged to overlapin the captured image Vd is estimated based on the movement prediction.Thus, the first hidden state determination unit 41 of the presentembodiment is configured to calculate, as the overlap ratio α, aproportion of the overlap area 55 to the imaged area 50 a of theoccupant 5 a at a far position who is predicted to be hidden by theoccupant 5 b in the standing posture close to the camera 8.

That is, as shown in FIG. 9 , the first hidden state determination unit41 of the present embodiment detects the imaged area 50 for each of theoccupants 5 imaged in the captured image Vd (step 101). Next, the firsthidden state determination unit 41 performs the movement prediction foreach of the occupants 5 (step 102). Subsequently, the first hidden statedetermination unit 41 estimates the overlap area 55 formed in the imagedarea 50 of each occupant 5 based on the movement prediction, andcalculates the overlap ratio α (step 103). Then, when the overlap ratioα is equal to or greater than the predetermined overlap determinationvalue αth (α≥αth, YES in step 104), the first hidden state determinationunit 41 of the present embodiment determines that the hidden stateoccurs in the captured image Vd (step 105).

(Number-of-Persons Difference Determination)

Next, detection and determination of the hidden state based onnumber-of-persons difference determination performed by the secondhidden state determination unit 42 will be described.

As shown in FIGS. 10 and 11 , in the hidden state detection unit 40 ofthe present embodiment, the second hidden state determination unit 42measures, based on analysis of the captured image Vd, the number ofentering and exiting persons H entering and exiting the vehicle interior6 that is the monitored space 11 imaged in the captured image Vd, thatis, the number of boarding and alighting occupants of the vehicle 1. Thesecond hidden state determination unit 42 of the present embodiment isconfigured to specify the total number of persons in the vehicleinterior 6 based on the measurement of the number of boarding andalighting occupants.

In detail, as shown in FIGS. 1 and 3 , in the monitoring system 15 ofthe present embodiment, the rear seat area A1, a front seat area A2, andan intermediate seat area A3, which are disposed in a substantiallyU-shape, are set in the vehicle interior 6 of the vehicle 1 imaged bythe camera 8. In the vehicle interior 6 of the vehicle 1, a floor areaA4 in which the occupant 5 can be on board in the standing posture isset in a range surrounded by the rear seat area A1, the front seat areaA2, and the intermediate seat area A3. Further, in the vehicle interior6 of the vehicle 1, a boarding and alighting area A5 where the occupant5 is prohibited to stay is set in the vicinity of the slide doors 4 and4 that open and close the door opening portion 3. When the occupant 5 isdetected in the boarding and alighting area A5, the second hidden statedetermination unit 42 of the present embodiment determines that theoccupant 5 is an occupant 5 on board the vehicle 1 or an occupant 5alighting from the vehicle 1.

Specifically, as shown in FIGS. 10 and 11 , the second hidden statedetermination unit 42 of the present embodiment specifies a movementdirection of the occupant 5 located in the boarding and alighting areaA5. Further, when the movement direction of the occupant 5 is adirection from the door opening portion 3 toward the inside of thevehicle interior 6 (see FIG. 10 , leftward in FIG. 10 ), the secondhidden state determination unit 42 determines that the occupant 5 is onboard the vehicle 1. When the movement direction of the occupant 5 is adirection from the inside of the vehicle interior 6 toward the dooropening portion 3 (see FIG. 11 , rightward in FIG. 11 ), the secondhidden state determination unit 42 determines that the occupant 5 is toalight from the vehicle 1.

In the image analysis apparatus 10 of the present embodiment, boardingand alighting determination for the occupant 5 performed by the secondhidden state determination unit 42 is performed in a state in which thecaptured image Vd of the vehicle interior 6 imaged by the camera 8 isconverted into a top view as shown in FIG. 3 . Thus, the second hiddenstate determination unit 42 of the present embodiment can measure thenumber N of boarding and alighting occupants of the vehicle 1 with highaccuracy.

Further, the second hidden state determination unit 42 of the presentembodiment measures the number of boarding and alighting occupants ofthe vehicle 1 by adopting “+1” when one occupant 5 on board the vehicle1 is detected, and adopting “−1” when one occupant 5 alighting from thevehicle 1 is detected. Thus, the second hidden state determination unit42 of the present embodiment can specify the total number of occupants 5located in the vehicle interior 6 based on the measurement of the numberof boarding and alighting occupants.

That is, as shown in FIG. 12 , the second hidden state determinationunit 42 of the present embodiment converts the captured image Vd of thevehicle interior 6 imaged by the camera 8 into the top view (step 201),and determines whether the occupant 5 is detected in the boarding andalighting area A5 before the door opening portion 3 (step 202). Next,when the occupant 5 is detected in the boarding and alighting area A5(YES in step 202), the second hidden state determination unit 42determines whether the occupant 5 is on board the vehicle 1 or alightsfrom the vehicle 1 (step 203 and step 204). Further, when the occupant 5on board the vehicle 1 is detected (YES in step 203), the second hiddenstate determination unit 42 adds “1” to the number N of boarding andalighting occupants of the vehicle 1 to be measured (N=N+1, step 205).When the occupant 5 alighting from the vehicle 1 is detected (YES instep 204), the second hidden state determination unit 42 subtract “1”from the number N of boarding and alighting occupants of the vehicle 1to be measured (N=N−1, step 206). Then, the second hidden statedetermination unit 42 of the present embodiment specifies the number Nof boarding and alighting occupants thus accumulated as the total numberNa of occupants 5 located in the vehicle interior 6 (Na=N, step 207).

The second hidden state determination unit 42 of the present embodimentcalculates the detected number Nd of occupants 5 imaged in the capturedimage Vd as the detected number of persons H imaged in the capturedimage Vd of the vehicle interior 6 based on the detection of theskeleton points SP as described above. Specifically, when the occupant 5whose main skeleton points SP used for the above posture determinationor the like can be extracted is detected in the captured image Vd of thevehicle interior 6 imaged by the camera 8, the second hidden statedetermination unit 42 of the present embodiment adds the occupant 5 tothe detected number Nd of persons in the captured image Vd. Further, thesecond hidden state determination unit 42 compares the total number Naof persons in the vehicle interior 6 specified by the measurement of thenumber N of boarding and alighting occupants as described above with thedetected number Nd of occupants 5 imaged in the captured image Vd. Thesecond hidden state determination unit 42 of the present embodiment isconfigured to perform the detection and determination of the hiddenstate occurring in the captured image Vd based on a difference betweenthe total number Na of persons and the detected number Nd of persons.

In more detail, as shown in FIG. 13 , when the second hidden statedetermination unit 42 of the present embodiment specifies the totalnumber Na of persons in the vehicle interior 6 based on the measurementof the number N of boarding and alighting occupants (step 301), thesecond hidden state determination unit 42 subsequently calculates thedetected number Nd of occupants 5 imaged in the captured image Vd (step302). Next, the second hidden state determination unit 42 calculates adifference value δ (δ=Na−Nd, step 303) between the total number Na ofpersons in the vehicle interior 6 specified based on the measurement ofthe number N of boarding and alighting occupants and the detected numberNd of occupants 5 imaged in the captured image Vd. Then, when thedifference value δ is a value larger than “0” (δ>0, YES in step 304),the second hidden state determination unit 42 of the present embodimentdetermines that the hidden state occurs in the captured image Vd (step305).

(Camera Proximity Position Determination)

Next, detection and determination of the hidden state based on cameraproximity position determination performed by the third hidden statedetermination unit 43 will be described.

As shown in FIG. 14 , when the occupant 5 in the vehicle interior 6stands at a position in proximity to the camera 8, the occupant 5 mayblock a field of view of the camera 8. Thus, the hidden state may occurin the captured image Vd in such a manner that the occupant 5 at theposition in proximity to the camera 8 hides another occupant 5 locatedin the vehicle interior 6.

In view of the above description, in the hidden state detection unit 40of the present embodiment, the third hidden state determination unit 43performs, at such a hiding position Px in proximity to the camera 8,detection and determination of the occupant 5 imaged in the capturedimage Vd. When the occupant 5 imaged in the captured image Vd isdetected at the hiding position Px, the third hidden state determinationunit 43 of the present embodiment regards that the hidden state occursin the captured image Vd.

Specifically, as shown in FIGS. 14 and 15 , the third hidden statedetermination unit 43 of the present embodiment calculates a size of animaged area 70 of the occupant 5 imaged in the captured image Vd (step401). Next, the third hidden state determination unit 43 calculates, asan image ratio β, a proportion of the imaged area 70 of the occupant 5to the entire captured image Vd (step 402). Subsequently, the thirdhidden state determination unit 43 compares the image ratio β with apredetermined proximity determination value βth (step 403). In the thirdhidden state determination unit 43 of the present embodiment, processingin steps 401 to 403 is performed for all the occupants 5 imaged in thecaptured image Vd. Further, when the occupant 5 having the image ratio βequal to or greater than the proximity determination value βth isdetected (β≥βth, YES in step 403), the third hidden state determinationunit 43 determines that the occupant 5 is imaged in the captured imageVd at the hiding position Px in proximity to the camera 8 (step 404).Thus, the third hidden state determination unit 43 of the presentembodiment regards that the hidden state occurs in the captured image Vd(step 405).

(Determination Output for Detection State of Skeleton Points)

Next, determination output for a detection state of the skeleton pointsSP in the captured image Vd in which the occurrence of the hidden stateis detected will be described.

The monitoring system 15 of the present embodiment is configured suchthat even when the occurrence of the hidden state is detected asdescribed above, the administrator 35 outside the vehicle, such as theoperator 33 stationed in the operation center 32 of the vehicle 1, canconfirm the captured image Vd of the vehicle interior 6 imaged by thecamera 8. At this time, in the monitoring system 15 of the presentembodiment, detection output of the hidden state from the hidden statedetection unit 40 is distributed to the operation center 32 in which theoperator 33 is stationed, together with the captured image Vd of thevehicle interior 6.

In detail, as shown in FIG. 7 , the hidden state detection unit 40 ofthe present embodiment includes a determination output unit 80 thatperforms, as the detection output, the determination output for thedetection state of the skeleton points SP in the captured image Vd inwhich the occurrence of the hidden state is detected, when theoccurrence of the hidden state is detected.

Specifically, when the hidden state occurring in the captured image Vdis detected based on the overlap ratio determination performed by thefirst hidden state determination unit 41, as the detection output, thedetermination output unit 80 of the present embodiment outputs thedetection state of the skeleton points SP using the captured image Vd asbeing “indeterminate”. When the hidden state occurring in the capturedimage Vd is detected based on the number-of-persons differencedetermination performed by the second hidden state determination unit42, as the detection output, the determination output unit 80 outputsthe detection state of the skeleton points SP using the captured imageVd as being “indeterminate”. When it is regarded that the hidden stateoccurs in the captured image Vd based on the camera proximity positiondetermination performed by the third hidden state determination unit 43,as the detection output, the determination output unit 80 of the presentembodiment outputs the detection state of the skeleton points SP usingthe captured image Vd as being “undeterminable”.

That is, output of “indeterminate” from the hidden state detection unit40 indicates a state in which accuracy for acquisition of the occupantinformation Ich based on the detection of the skeleton points SPdecreases due to the presence of an undetected occupant 5 whose skeletonpoints SP cannot be detected. Output of “undeterminable” from the hiddenstate detection unit 40 indicates a state in which it is not possible todetermine whether the skeleton points SP are undetected even by theoverlap ratio determination performed by the first hidden statedetermination unit 41 and the number-of-persons difference determinationperformed by the second hidden state determination unit 42. Further, inthe monitoring system 15 of the present embodiment, the output of“undeterminable” is handled as “occurrence of the abnormality” with highemergency, as in a case of detection output of the abnormality from theabnormality detection unit 22. Thus, the monitoring system 15 of thepresent embodiment is configured such that the administrator 35 outsidethe vehicle can confirm the captured image Vd of the vehicle interior 6while referring to the detection state of the skeleton points SP, thatis, accuracy of the occupant information Ich acquired by the imageanalysis.

In detail, as shown in FIG. 16 , when the image analysis apparatus 10 ofthe present embodiment acquires the captured image Vd of the inside ofthe vehicle interior 6 (step 501), the image analysis apparatus 10performs the image analysis to detect the skeleton points SP of theoccupant 5 imaged in the captured image Vd (step 502). Next, the hiddenstate detection unit 40 of the image analysis apparatus 10 performshidden state determination (step 503). Then, when the hidden state inthe captured image Vd is detected by the hidden state determination (YESin step 504), the image analysis apparatus 10 performs detection outputcontrol of distributing, to the administrator 35 outside the vehicle,the detection output of the hidden state together with the capturedimage Vd (step 505).

Specifically, as shown in FIG. 17 , in the image analysis apparatus 10of the present embodiment, first, the third hidden state determinationunit 43 of the hidden state detection unit 40 performs the detection anddetermination of the hidden state, that is, the camera proximityposition determination (step 601). Then, when the occupant 5 imaged inthe captured image Vd is detected at the hiding position Px in proximityto the camera 8 (YES in step 602), the determination output unit 80 ofthe hidden state detection unit 40 of the present embodiment outputs“undeterminable” (step 603).

When the occupant 5 is not detected at the hiding position Px inproximity to the camera 8 (NO in step 602), the first hidden statedetermination unit 41 of the hidden state detection unit 40 performs thedetection and determination of the hidden state, that is, the overlapratio determination (step 604). Subsequently, the second hidden statedetermination unit 42 of the hidden state detection unit 40 performs thedetection and determination of the hidden state, that is, thenumber-of-persons difference determination (step 605). Further, when thehidden state in the captured image Vd is detected by the overlap ratiodetermination and the number-of-persons difference determination (YES instep 606), the hidden state detection unit 40 determines that theoccupant 5 whose skeleton points SP are undetected is present in thevehicle interior 6 (step 607). Thus, the image analysis apparatus 10 ofthe present embodiment is configured such that the determination outputunit 80 of the hidden state detection unit 40 outputs “indeterminate” asthe detection output of the hidden state (step 608).

As shown in FIG. 16 , when the hidden state in the captured image Vd isnot detected in the hidden state determination (NO in step 504), theabnormality detection unit 22 of the image analysis apparatus 10performs detection and determination of the abnormality in the vehicleinterior 6 imaged in the captured image Vd (step 506). That is, asdescribed above, in the image analysis apparatus 10 of the presentembodiment, the detection and determination of the abnormality performedby the abnormality detection unit 22 is based on the posturedetermination for the occupant 5 performed by the posture determinationunit 26 based on the detection of the skeleton points SP, specifically,detection of the falling posture. The image analysis apparatus 10 of thepresent embodiment is configured such that when the abnormality in thevehicle interior 6 is detected by the abnormality detection unit 22 (YESin step 507), the abnormality detection unit 22 distributes, to theadministrator 35 outside the vehicle, the detection output of theabnormality together with the captured image Vd (step 508).

Next, actions of the present embodiment will be described.

That is, in the image analysis apparatus 10 of the present embodiment,the overlap ratio α of the imaged area 50 estimated based on themovement prediction is calculated for each of the occupants 5 imaged inthe captured image Vd of the vehicle interior 6. The difference value δbetween the total number Na of persons in the vehicle interior 6specified by the measurement of the number N of boarding and alightingoccupants and the detected number Nd of the occupants 5 imaged in thecaptured image Vd is calculated. Further, the detection anddetermination as to whether the occupant 5 imaged in the captured imageVd is present at the hiding position Px in proximity to the camera 8 isperformed. The occurrence of the hidden state in which the plurality ofoccupants 5 are imaged to overlap in the captured image Vd is detectedbased on results of the overlap ratio determination, thenumber-of-persons difference determination, and the camera proximityposition determination.

Next, effects of the present embodiment will be described.

(1) The image analysis apparatus 10 includes the person informationacquisition unit 25 that acquires the information Ih of the person Himaged in the captured image Vd, that is, the occupant information Ichof the occupant 5 by setting the vehicle interior 6 of the vehicle 1imaged by the camera 8 as the monitored space 11 and analyzing thecaptured image Vd. Further, the image analysis apparatus 10 includes thehidden state detection unit 40 that detects the occurrence of the hiddenstate in which the plurality of occupants 5 are imaged to overlap in thecaptured image Vd.

According to the above configuration, it is possible to detect thehidden state occurring in the captured image Vd and grasp the state inwhich the occupant 5 in the vehicle interior 6 is not correctly imagedin the captured image Vd. In this case, for example, informationacquisition with high accuracy can be ensured by avoiding analytical useof the captured image Vd in which the hidden state occurs.

(2) The first hidden state determination unit 41 provided in the hiddenstate detection unit 40 has a function as a movement prediction unit 90a that performs the movement prediction for each of the occupants 5imaged in the captured image Vd. In addition, the first hidden statedetermination unit 41 has a function as an overlap ratio calculationunit 90 b that calculates the overlap ratio α for the imaged area 50 ofeach of the occupants 5, and the imaged area 50 is estimated to beimaged to overlap in the captured image Vd based on the movementprediction. The first hidden state determination unit 41 performs thedetection and determination of the hidden state based on the comparisonbetween the overlap ratio α and the overlap determination value αth.

According to the above configuration, the occurrence of the hidden statein the captured image Vd can be detected with high accuracy. Further,the image analysis apparatus 10 has an advantage that the occurrence ofthe hidden state can be predicted in advance. Thus, the informationacquisition can be performed with higher accuracy.

(3) The second hidden state determination unit 42 provided in the hiddenstate detection unit 40 has a function as anumber-of-entering-and-exiting-persons measurement unit 90 c thatmeasures the number N of boarding and alighting occupants 5 that is thenumber of entering and exiting persons H entering and exiting themonitored space 11. The second hidden state determination unit 42performs the detection and determination of the hidden state based onthe difference between the total number Na of persons in the vehicleinterior 6 specified by the measurement of the number N of boarding andalighting occupants and the detected number Nd of occupants 5 imaged inthe captured image Vd.

That is, when the hidden state does not occur in the captured image Vd,the detected number Nd of occupants 5 imaged in the captured image Vd isequal to the total number Na of persons in the vehicle interior 6.Therefore, according to the above configuration, the hidden stateoccurring in the captured image Vd can be easily detected with a simpleconfiguration.

(4) The third hidden state determination unit 43 provided in the hiddenstate detection unit 40 regards that the hidden state occurs in thecaptured image Vd when the occupant 5 imaged in the captured image Vd isdetected at the hiding position Px in proximity to the camera 8.

That is, the hidden state may occur in the captured image Vd in such amanner that the occupant 5 imaged in the captured image Vd at the hidingposition Px in proximity to the camera 8 blocks the field of view of thecamera 8 to hide another occupant 5 located in the vehicle interior 6.In this case, it is not possible to determine whether the hidden stateactually occurs. In view of the above description, as in the aboveconfiguration, it is regarded that the hidden state occurs. In thiscase, for example, the information acquisition with high accuracy can beensured by avoiding analytical use of the captured image Vd in which thehidden state occurs.

(5) The third hidden state determination unit 43 has a function as animage ratio calculation unit 90 d that calculates, as the image ratio β,the proportion of the imaged area 70 of the occupant 5 imaged in thecaptured image Vd to the entire captured image Vd. The third hiddenstate determination unit 43 has a function as a camera proximityposition determination unit 90 e that determines that the occupant 5 isimaged in the captured image Vd at the hiding position Px when theoccupant 5 having the image ratio β equal to or greater than thepredetermined proximity determination value βth is detected. Thus, theoccupant 5 imaged in the captured image Vd at the hiding position Px inproximity to the camera 8 can be detected with a simple configuration.

(6) The image analysis apparatus 10 includes the skeleton pointdetection unit 23 that detects the skeleton points SP of the occupant 5included in the captured image Vd. The image analysis apparatus 10includes the abnormality detection unit 22 that detects the abnormalityoccurring in the vehicle interior 6 based on the occupant informationIch acquired by detecting the skeleton points SP.

That is, by detecting the skeleton points SP, the physical occupantinformation Ich such as a posture and a physique of the occupant 5 canbe acquired with high accuracy. Thus, based on the acquired occupantinformation Ich, it is possible to perform, with high accuracy, thedetection and determination of the abnormality for the vehicle interior6 in which the occupant 5 is on board. However, when the hidden stateoccurs in the captured image Vd, the detection state of the skeletonpoints SP also deteriorates. As a result, the detection anddetermination of the abnormality may not be performed with highaccuracy. Therefore, a more remarkable effect can be obtained byapplying the detection and determination of the hidden state shown inthe above (1) to (5) to such a configuration.

(7) The hidden state detection unit 40 includes the determination outputunit 80 that performs, as the detection output of the hidden state, thedetermination output for the detection state of the skeleton points SPin the captured image Vd in which the occurrence of the hidden state isdetected, when the occurrence of the hidden state is detected.

According to the above configuration, it is possible to correctly graspthe detection state of the skeleton points SP changed due to theoccurrence of the hidden state. Thus, it is possible to appropriatelyuse the captured image Vd in which the hidden state occurs.

The above embodiment can be modified and implemented as follows. Theabove embodiment and the following modifications can be implemented incombination with each other as long as the embodiment and themodifications are technically not in conflict with each other.

In the above embodiment, the infrared camera is used as the camera 8,but a model thereof may be appropriately changed. For example, a visiblelight camera or the like may be used.

In the above embodiment, the imaged area 60 of the torso portion is usedas the imaged area 50 of the person H in the calculation of the overlapratio α, but for example, a range of the imaged area 50 used for thecalculation of the overlap ratio α may be appropriately changed, such asincluding a head. In addition, a future time at which the movementprediction and overlap estimation are performed may be appropriatelychanged. A specific method for the movement prediction may beappropriately changed.

In the above embodiment, the boarding and alighting determination forthe occupant 5 is performed with the captured image Vd converted intothe top view, but conversion to the top view may not necessarily beperformed. For example, when the total number Na of persons in thevehicle interior 6 can be acquired by a method other than the imageanalysis, such as boarding reservation information, a value in theboarding reservation information may be used.

In the above embodiment, the detected number Nd of occupants 5 imaged inthe captured image Vd is calculated based on the detection of theskeleton points SP, specifically, an extraction possibility of the mainskeleton points SP, but may not necessarily be based on the detection ofthe skeleton points SP. The number of occupants 5 recognized in thecaptured image Vd by using other methods may be used as the detectednumber Nd of persons. The difference value δ between the total number Naof persons in the vehicle interior 6 and the detected number Nd ofoccupants 5 may not necessarily be calculated, and match determinationmay be made simply.

In the above embodiment, the proportion of the imaged area 70 of theoccupant 5 imaged in the captured image Vd to the entire captured imageVd is set as the image ratio β, and it is determined that the occupant 5having the image ratio β equal to or greater than the predeterminedproximity determination value βth is imaged in the captured image Vd atthe hiding position Px. However, the disclosure is not limited thereto,and the detection and determination of the occupant 5 imaged in thecaptured image Vd at the hiding position Px may be appropriatelychanged. That is, as long as it can be specified that when the occupant5 in the vehicle interior 6 stands at the position in proximity to thecamera 8, the occupant 5 blocks the field of view of the camera 8, thespecified position of the occupant 5 is the hiding position Px. Forexample, the image ratio β may not necessarily be calculated, and thecamera proximity position determination may be performed by acombination of the size of the imaged area 70 and a use state of anaccessory of the vehicle 1 such as the hanging strap or the handrailindicating that the occupant 5 is in the position in proximity to thecamera 8.

In the above embodiment, the posture determination for the occupant 5 isperformed based on the detection of the skeleton points SP. Theabnormality in the vehicle interior 6 imaged in the captured image Vd isdetected by detecting the falling posture. However, the disclosure isnot limited thereto, and the detection and determination of theabnormality may be performed by using other occupant information Ichacquired by the image analysis of the captured image Vd. Further, theacquisition of the occupant information Ich may be performed regardlessof the detection of the skeleton points SP. The acquired occupantinformation Ich may also be used for applications other than thedetection and determination of the abnormality.

In the above embodiment, the monitoring system 15 is formed byconnecting the plurality of information processing apparatuses 30, whichare disposed inside and outside the vehicle 1, with one another via theinformation communication network (not illustrated). In the imageanalysis apparatus 10, the in-vehicle information processing apparatus30 a mounted in the vehicle 1, and the information processing apparatus30 b which is outside the vehicle and constitutes the cloud server 31perform the image analysis processing in the distributed manner.However, a system configuration of the monitoring system 15 is notlimited thereto, and may be appropriately changed. For example, theimage analysis apparatus 10 may be mounted in the in-vehicle informationprocessing apparatus 30 a mounted in the vehicle 1. Further, theinformation processing apparatus 30 b, which is outside the vehicle andconstitutes the cloud server 31, may be disposed in the operation center32 of the vehicle 1 in which the operator 33 as the administrator 35 isstationed.

Further, for the captured image Vd of the inside of the vehicle interior6 confirmed by the administrator 35 when the abnormality occurs or thehidden state is detected, the captured image Vd imaged by the camera 8may be constantly distributed to the administrator 35 outside thevehicle, or may be distributed to the administrator 35 only when anevent occurs.

In the above embodiment, the monitoring system 15 is embodied with themonitored space 11 set as the vehicle interior 6 imaged by the camera 8in the vehicle 1. However, the monitored space 11 is not limitedthereto, and may be a room interior of a building. For example, themonitored space 11 may also be set outdoors.

According to an aspect of this disclosure, an image analysis apparatusincludes: a person information acquisition unit configured to analyze acaptured image of a monitored space imaged by a camera so as to acquireinformation of a person imaged in the captured image; and a hidden statedetection unit configured to detect occurrence of a hidden state inwhich a plurality of the persons are imaged to overlap in the capturedimage.

According to the above configuration, it is possible to detect thehidden state occurring in the captured image and grasp a state in whichthe person in the monitored space is not correctly imaged in thecaptured image. In this case, for example, information acquisition withhigh accuracy can be ensured by avoiding analytical use of the capturedimage in which the hidden state occurs.

According to the above aspect of the disclosure, it is preferable thatthe image analysis apparatus further includes: a movement predictionunit configured to perform movement prediction for each of the personsimaged in the captured image; and an overlap ratio calculation unitconfigured to calculate an overlap ratio for an imaged area of each ofthe persons, the imaged area being estimated, based on the movementprediction, to be imaged to overlap. It is preferable that the hiddenstate detection unit is configured to perform detection anddetermination of the hidden state based on a comparison between theoverlap ratio and an overlap determination value.

According to the above configuration, the occurrence of the hidden statein the captured image can be detected with high accuracy. Further, theimage analysis apparatus has an advantage that the occurrence of thehidden state can be predicted in advance. Thus, the informationacquisition can be performed with higher accuracy.

According to the above aspect of the disclosure, it is preferable thatthe image analysis apparatus further includes anumber-of-entering-and-exiting-persons measurement unit configured tomeasure the number of entering and exiting persons entering and exitingthe monitored space, and the hidden state detection unit is configuredto perform detection and determination of the hidden state based on adifference between the total number of persons in the monitored spacespecified by measuring the number of entering and exiting persons andthe detected number of persons imaged in the captured image.

That is, when the hidden state does not occur in the captured image, thedetected number of persons imaged in the captured image is equal to thetotal number of persons in the monitored space. Therefore, according tothe above configuration, the hidden state occurring in the capturedimage can be easily detected with a simple configuration.

According to the above aspect of the disclosure, it is preferable thatin the image analysis apparatus, the hidden state detection unit isconfigured to, when the person imaged in the captured image is detectedat a hiding position in proximity to the camera, regard that the hiddenstate occurs.

That is, the hidden state may occur in the captured image in such amanner that the person imaged in the captured image at the hidingposition in proximity to the camera blocks a field of view of the camerato hide another person located in the monitored space. In this case, itis not possible to determine whether the hidden state actually occurs.In view of the above description, as in the above configuration, it isregarded that the hidden state occurs. In this case, for example, theinformation acquisition with high accuracy can be ensured by avoidinganalytical use of the captured image in which the hidden state occurs.

According to the above aspect of the disclosure, it is preferable thatthe image analysis apparatus further includes: an image ratiocalculation unit configured to calculate, as an image ratio, aproportion of an imaged area of the person to an entire captured image;and a camera proximity position determination unit configured to, whenthe person having the image ratio equal to or greater than apredetermined proximity determination value is detected, determine thatthe person is imaged in the captured image at the hiding position.

According to the above configuration, the person imaged in the capturedimage at the hiding position in proximity to the camera can be detectedwith a simple configuration.

According to the above aspect of the disclosure, it is preferable thatthe image analysis apparatus further includes: a skeleton pointdetection unit configured to detect a skeleton point of the personincluded in the captured image; and an abnormality detection unitconfigured to detect an abnormality occurring in the monitored spacebased on information of the person acquired by detecting the skeletonpoint.

That is, by detecting the skeleton point, physical information such as aposture and a physique of the person can be acquired with high accuracy.Thus, detection and determination of the abnormality in the monitoredspace can be performed with high accuracy based on the acquiredinformation of the person. However, when the hidden state occurs in thecaptured image, a detection state of the skeleton point alsodeteriorates. As a result, the detection and determination of theabnormality may not be performed with high accuracy. Therefore, a moreremarkable effect can be obtained by applying the detection anddetermination of the hidden state shown in any of the aboveconfigurations to such a configuration.

According to the above aspect of the disclosure, it is preferable thatthe image analysis apparatus further includes a determination outputunit configured to, when the occurrence of the hidden state is detected,perform, as detection output of the hidden state, determination outputfor a detection state of the skeleton point in the captured image inwhich the occurrence of the hidden state is detected.

According to the above configuration, it is possible to correctly graspthe detection state of the skeleton point changed due to the occurrenceof the hidden state. Thus, it is possible to appropriately use thecaptured image in which the hidden state occurs.

According to the above aspect of the disclosure, it is preferable thatin the image analysis apparatus, the monitored space is a vehicleinterior of a vehicle, and the person is an occupant in the vehicle.

According to the above configuration, the information acquisition can beperformed with high accuracy for the occupant in the vehicle interior.

According to another aspect of this disclosure, a monitoring systemincludes the image analysis apparatus according to any one of the aboveaspects.

According to this disclosure, it is possible to grasp the state in whichthe person in the monitored space is not correctly imaged in thecaptured image.

The principles, preferred embodiment and mode of operation of thepresent invention have been described in the foregoing specification.However, the invention which is intended to be protected is not to beconstrued as limited to the particular embodiments disclosed. Further,the embodiments described herein are to be regarded as illustrativerather than restrictive. Variations and changes may be made by others,and equivalents employed, without departing from the spirit of thepresent invention. Accordingly, it is expressly intended that all suchvariations, changes and equivalents which fall within the spirit andscope of the present invention as defined in the claims, be embracedthereby.

What is claimed is:
 1. An image analysis apparatus comprising: a personinformation acquisition unit configured to analyze a captured image of amonitored space imaged by a camera so as to acquire information of aperson imaged in the captured image; and a hidden state detection unitconfigured to detect occurrence of a hidden state in which a pluralityof the persons are imaged to overlap in the captured image.
 2. The imageanalysis apparatus according to claim 1, further comprising: a movementprediction unit configured to perform movement prediction for each ofthe persons imaged in the captured image; and an overlap ratiocalculation unit configured to calculate an overlap ratio for an imagedarea of each of the persons, the imaged area being estimated, based onthe movement prediction, to be imaged to overlap, wherein the hiddenstate detection unit is configured to perform detection anddetermination of the hidden state based on a comparison between theoverlap ratio and an overlap determination value.
 3. The image analysisapparatus according to claim 1, further comprising: anumber-of-entering-and-exiting-persons measurement unit configured tomeasure the number of entering and exiting persons entering and exitingthe monitored space, wherein the hidden state detection unit isconfigured to perform detection and determination of the hidden statebased on a difference between the total number of persons in themonitored space specified by measuring the number of entering andexiting persons and the detected number of persons imaged in thecaptured image.
 4. The image analysis apparatus according to claim 1,wherein the hidden state detection unit is configured to, when theperson imaged in the captured image is detected at a hiding position inproximity to the camera, regard that the hidden state occurs.
 5. Theimage analysis apparatus according to claim 4, further comprising: animage ratio calculation unit configured to calculate, as an image ratio,a proportion of an imaged area of the person to an entire capturedimage; and a camera proximity position determination unit configured to,when the person having the image ratio equal to or greater than apredetermined proximity determination value is detected, determine thatthe person is imaged in the captured image at the hiding position. 6.The image analysis apparatus according to claim 1, further comprising: askeleton point detection unit configured to detect a skeleton point ofthe person included in the captured image; and an abnormality detectionunit configured to detect an abnormality occurring in the monitoredspace based on information of the person acquired by detecting theskeleton point.
 7. The image analysis apparatus according to claim 6,further comprising: a determination output unit configured to, when theoccurrence of the hidden state is detected, perform, as detection outputof the hidden state, determination output for a detection state of theskeleton point in the captured image in which the occurrence of thehidden state is detected.
 8. The image analysis apparatus according toclaim 1, wherein the monitored space is a vehicle interior of a vehicle,and the person is an occupant in the vehicle.
 9. A monitoring system,comprising the image analysis apparatus according to claim 1.